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What is claimed is: 




1 . A method for querying stored multimedia data in a computer system, 
comprising: 

receiving into an intermediate level a hj£h-level concept from a user 
describing data to be retrieved; 

translating, in said intermediate leveA, said high-level concept into 
low-level queries by using system pre-defined high-level concepts; and 

transferring said low-level queries to a low level comprising one or more 
search engines; said one or more search engines performing a query of the stored 
multimedia information using said low-leve/ queries. 



2. The method of claim 1 wnerein said intermediate level comprises: 
a set of library modules? said set of library modules comprising: 
a concept library moddle for storing concepts; 

one or more library/nodules adapted to store said data from said one or 
more data sources; 

a cataloger module adapted to construct a new concept from said 
high-level concept using darca from said concept library and library modules, thereby 
creating a concept construct, and to pass said concept construct to said concept library 
module for storage as A concept; and 
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an interpreter module adapted to translate said high-level concqrt into 
low-level queries using said concepts stored in said construct library and y to pass said 
low-level queries to said one or more search engines. 

3. The method of claim 2 wherein said set of library modules further 
comprises at least one library module selected from the group comprising: 

a feature library module adapted to store nailtimedia features; 
a matching algorithm library module ajfepted to store matching 
algorithms; and 

a constraint library module adapted to store feature constraints. 

4. The method of claim 3 wherein each said library module further 
comprises an application program interface to receive said data from a said data source. 



5. The method of claim 3 wherein said cataloger module further performs the 



steps of: 



child-concepts 



selecting af set of concept features from said feature library module; 
selecti*4 a set of concepts from said concept library module for use as 

; 

electing a set of constraints on said child concepts from said constraint 



library module 
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6. The method of claim 1 wherein said each said concept comprises a trivet 
of a set of child-concepts, a set of features, and a set of relationships. 

7. The method of claim 6 wherein said concepts comprise a^iierarchical 
fuzzy graph data tree-structure comprising nodes, aggregation edges,yand association 
edges and wherein: 

said nodes correspond to said concepts and said/features; 
said aggregation edges correspond to parenlwhild relationships; and 
said association edges correspond to sai/ constraints. 

8. The method of claim 7 wherein said edges are weighted. 



9. The method of claim 3 fucmer comprising a matching algorithm 
comprising GetNextMatch(), Assignl^6xtMatch(), and ShiftNextMatch() procedures, 
wherein: 

said GetNextMat6h() procedure comprises the steps: 
testqueue: /if queue. Empty (); 

return NULL; 
head --> queue.PopQ; 
if /*eai.Complete(); 

return/head; 
head! — > Aead.CopyQ; 
/zeat/2.^ssignNextMatch(); 

/ 
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if head2.V alidQ; 



# 



queue. P\ish(head2); 
Aea</.ShiftNextMatch(); 
queue. Fu$h(head); 
5 Goto testqueue; 

said AssignNextMatch() procedure comprises the steps: 
child --> GetNdxtUnassigned(); 
child.match _ptrt±+\ 
if {child.matchlptr} ==NULL), then; 
p 10 child.match _ptr --> cMrf.Get^extMatch(); 

y3 Make child an assigned node; 

m said ShiftNextMatch() procedure comprises the steps: 

CTn/rf -> Ge|sfextUnassigned(); 
L child.match /pfr++; 

[ 15 if (child.match j)tr = M/LL), thdh; 

child.match _ptr --> c/nWjbetNextMatch(); 

wherein variables Aea<£ Head2, and c/u7</ f all correspond to concept nodes; 
variable #wewe denotes a prioritM queue of the corresponding concept node; and 
match _ptr is a pointer to the next possible match for a given concept node; Pop() is a 
20 method to get the next node off the priority queue; Push() is a method to put a node on 

the priority queue; Empty() is a method to check if the priority queue is empty; Copy() is 
a method to copy a node; CdmpleteQ is a method to check if the children assignment is 
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complete; Valid() is a method to check if the children assignment meets the constraints; 
and GetNextUnassignedQ is a method to select a variable that is unassignec 



10. A matching algorithm comprising GetNextMatch(), ^signNextMatch(), 
and ShiftNextMatch() procedures, wherein: 

said GetNextMatch() procedure comprises the ^teps: 
testqueue: if queue. Empty (); 

return NULL; 
head — > queue.PopQ; 
if /*ea<i.Complete(); 

return head; 
head2 -> heaa.CopyQ; 
Aea£/2.AssisnNextMatch(); 
if head2. J alidQ; 

queue. Push(head2)\ 
Ae^.ShiftNextMatch(); 
queue. Push(head) ; 
Goto testqueue; 

said AssignNextMatfch() procedure comprises the steps: 
child — > GetNextUnassigned(); 
ihild.match jttfrH-; 
/ if (child.match j)tr) = NULL), then; 



child.matclf j>tr --> c/»W.GetNextMatch(); 
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Make child an assigned node; 

said ShiftNepctMatch() procedure comprises the steps: 
Child --> GetNextUnassigned(); 
{child.match J9/H-+; 
if {child.match j>tr ==]NULL\ then; 

child.match j?tA~> cM<i.GetNextMatch(); 

wherein variables head, head2, and child, all correspond to. concept nodes; 
variable queue denotes a priority queue of the corresponding concept node; and 
match _ptr is a pointer to the next possible match for a given concept node; Pop() is a 
method to get the next node off the priority queue; Push() is a method to put a node on 
the priority queue; Empty()\is a method to check if the priority queue is empty; Copy() is 
a method to copy a node; Carnplete() is a method to check if the children assignment is 
complete; Valid() is a methodtto check if the children assignment meets the constraints; 
and GetNextUnassignedQ is a method to select a variable that is unassigned. 



11. A program storage\deyice readable by machine, tangibly embodying a 
program of instructions executable by the machine to perform method steps for querying 
stored multimedia data, said method\steps comprising: 

receiving into an intermediate level a high-level concept from a user 
describing data to be retrieved; 

translating, in said intermediate level, said high-level concept into 
low-level queries by using system pre-defined high-level concepts; 
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transferring said low-level queries to a low level comprising one/6r more 
search engines; said one or more search engines performing a query of the stored 
multimedia information using said low-level queries. j 

I 

12. The apparatus of claim 1 1 wherein said intermediate lefVel comprises: 
a set of library modules, said set of library modules Comprising; 
a concept library module for storing concepts; anc 
one or more library modules adapted to store s^fd data from said one or 

more data sources; 

a cataloger module adapted to construct a n£w concept from said 
high-level concept using data from said concept library ^nd library modules, thereby 
creating a concept construct, and to pass said concept/construct to said concept library 
module for storage as a concept; and 

an interpreter module adapted to translate said high-level concept into 
low-level queries using said concepts stored in said construct library and to pass said 
low-level queries to said one or more searcb^ngines. 

13. The apparatus of claim 12 wherein said set of library modules further 
comprises at least one library modulp selected from the group comprising: 

a feature library module adapted to store multimedia features; 
a matching algo/ithm library module adapted to store matching 
algorithms; and 

a constraint library module adapted to store feature constraints. 

/ 
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14. The apparatus of claim 13 wherein each said library module fuller 
comprises an application program interface to receive said data from a saicj/data source. 



15. The apparatus of claim 13 wherein said cataloger mo further performs 
the steps of: 

selecting a set of concept features from said featucfe library module; 

selecting a set of concepts from said concept Hilary module for use as 
child-concepts; and 

selecting a set of constraints on said child concepts from said constraint 
library module. 

16. The apparatus of claim 1 1 wherein said each said concept comprises a 
triplet of a set of child-concepts, a set of features/ and a set of relationships. 

17. The apparatus of claim 16 wljerein said concepts comprise a hierarchical 
fuzzy graph data tree-structure comprisirjg nodes, aggregation edges, and association 
edges and wherein: 

said nodes correspondfto said concepts and said features; 
said aggregation edges correspond to parent-child relationships; and 
said association e^ges correspond to said constraints. 



18. The apparatus of claim 17 wherein said edges are weighted. 
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19. The apparatus of claim 13 further comprising a matching algorithm 
comprising GetNextMatch(), AssignNextMatch(), and ShiftN<^tMatch() procedures, 
wherein: 

said GetNextMatch() procedure comprised the steps: 
testqueue: if queue.Emptyof 
return NULL; / 
A 

if head. Completefr); 
return hlad; 
JdX 

/iea</2AssjgnNextMatch(); 
/a 



head — > queue.Pop( 



10 /*ea</2 — > head.CopyQ; 



if head2.fJd\\<\Q\ 

queue. Push(head2) ; 
/^.ShiftNextMatch(); 
15 queue. Push(head); 

Goto testqueue; 

said AssignNextMatch(jf procedure comprises the steps: 
ctoW --> GetNextUnassigned(); 
child.match j>tr++\ 
20 if {child.match j)tr) = NULL), then; 

child.match _ptr-J> c/n7rf.GetNextMatch(); 
Make c/iiW an assignecynode; 

said ShiftNextMatch() procedure comprises the steps: 

/ 
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Child — > GetNextpnassigned(); 
child.match _ptr^-; 
if (child.match j>tr == NULL), then; 

child.match _ptr —> c/uW.GetftextMatch(); 
wherein variables head, headE, and child, all correspond to concept nodes; 
variable queue denotes a priority queue of the corresponding concept node; and 
match _ptr is a pointer to the next possible match for a given concept node; Pop() is a 
method to get the next node off the priority queue; Push() is a method to put a node on 
the priority queue; Empty() is a method to check if the priority queue is empty; Copy() is 
a method to copy a node; Complete() is a method to check if the children assignment is 
complete; Valid() is a method to/check if the children assignment meets the constraints; 
and GetNextUnassignedQ is a method to select a variable that is unassigned. 
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